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LONG-TERM  GOALS 

The  Earth  System  Prediction  Capability  (ESPC)  interagency  program  was  established  in  2010  as  an 
effort  to  improve  collaboration  across  the  federally  sponsored  environmental  research  and  operational 
prediction  communities.  The  ESPC  goal  is  to  coordinate  scientific  development  and  operational 
implementation  of  improved  global  prediction  at  the  weather  to  climate  interface.  Part  of  the  ESPC 
role  is  to  advocate  for  and  support  the  basic  foundations  of  modeling  and  data  systems,  such  as  the 
computational  efficiency  of  models  and  common  coupled  system  architectures. 

The  Optimized  Infrastructure  for  ESPC  {01 for  ESPC)  project  is  working  to  advance  the  computational 
foundations  of  ESPC  through  three  integrative  activities.  These  activities  are  designed  to  leverage 
numerical  libraries  across  agencies;  to  explore  the  impacts  of  emerging  computing  platforms  on 
coupled  model  infrastructure;  and  to  reconcile  conventions  across  different  model  architectures.  The 
activities  are  serving  as  a  starting  point  for  a  broader  effort  funded  through  the  2013  NOPP  program 
Advancing  Air-Ocean-Land-Ice  Global  Coupled  Prediction  on  Emerging  Computational  Architectures 
(AOEI). 

OBJECTIVES 

1,  Update  Numerical  Libraries  in  ESMF.  The  Earth  System  Modeling  Eramework  (ESME)  is 
widely  used  in  the  ESPC  community  for  building  and  coupling  models.  We  will  adapt 
MOAB,  a  finite  element  mesh  package  in  active  development  at  the  University  of  Wisconsin 
and  Argonne  National  Laboratory,  for  use  as  an  ESMF  computational  kernel.  The  intent  is  to 
evaluate  MOAB  as  a  replacement  for  the  native  ESMF  implementation  of  finite  element  mesh 
software.  The  framework  uses  this  mesh  software  to  create  a  unified  representation  of 
structured  and  unstructured  grids,  and  define  operations  that  work  across  grid  types.  The 
desired  outcome  is  an  immediate  boost  of  new  features  in  ESMF,  and  in  the  longer  term,  an 
opportunity  to  leverage  the  work  that  goes  into  ongoing  functions  and  optimizations  in  MOAB 
and  associated  DOE  libraries. 

2,  Optimize  Component  Architecture  for  GPUs.  We  will  prototype  a  set  of  mechanisms  that 
enable  a  component  framework  to  optimize  for  accelerators,  and  explore  how  accelerators  may 
affect  mapping  of  components  to  resources.  The  product  will  be  a  code  consisting  of  sample  or 
toy  components  with  representative  operations.  The  intent  of  this  research  is  to  inform  both 
infrastructure  development  and  optimizations  undertaken  by  ESPC  modeling  teams. 
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3.  Couple  HYCOM  to  CESM.  We  will  work  with  developers  of  the  HYbrid  Coordinate  Ocean 
Model  (HYCOM)  to  couple  it  to  the  Community  Earth  System  Model  (CESM).  This  will 
enable  HYCOM  to  be  evaluated  in  many  new  coupled  configurations.  The  activity  will  also 
define  a  mapping  between  the  different  coupling  rules  and  conventions  implemented  in 
HYCOM  and  CESM.  HYCOM  uses  a  standard  ESME-based  interface  developed  by  a  NOAA 
and  DoD  consortium  (the  National  Unified  Operational  Prediction  Capability,  or  NUOPC 
conventions),  while  CESM  has  its  own  native  format  based  on  the  Model  Coupling  Toolkit 
(MCT).  This  is  an  important  exercise  because  that  mapping  will  apply  not  just  to  this  coupling, 
but  will  provide  insight  into  how  any  components  that  use  the  NOAA/DoD  interface  can  be 
coupled  into  a  CESM-based  system,  and  vice  versa. 

APPROACH 

Team  members  hold  weekly  telecons  to  discuss  plans  and  progress.  Participants  include  both  technical 

and  scientific  team  members,  below: 

Argonne  National  Eaboratory/University  of  Chicago/University  of  Wisconsin:  Rob  Jacob,  Jayesh 
Krishna,  Tim  Tautges 

Elorida  State  University:  Alexandra  Bozec,  Eric  Chassignet 

NCAR:  Jim  Edwards,  Mariana  Vertenstein,  Tony  Craig 

NASA  Jet  Propulsion  Laboratory:  Peggy  Li 

Naval  Research  Laboratory:  Alan.Wallcraft 

University  of  C A  San  Diego:  Caroline  Papadopoulos,  Julie  McClean 

University  of  Colorado/NOAA:  Cecelia  DeLuca,  Kathy  Saint,  Gerhard  Theurich,  Eei  Liu,  Bob 
Oehmke,  Walter  Spector 

University  of  Miami:  Ben  Kirtman 


Above  are  subscribed  to  an  espc_tech@list.woc.noaa.gov  mailing  list  used  for  organizing  the  calls. 

A  website  for  the  01 for  ESPC  project  was  created  on  the  CoG  collaboration  environment: 
http://www.earthsystemcog.org/projects/espc-infrastructure/ 

Some  of  the  01  for  ESPC  project  activities  are  closely  associated  with  a  Coupling  Testbed,  which  is  a 
collaborative  area  for  groups  working  on  research  problems  involving  model  coupling.  The  Coupling 
Testbed  was  funded  under  the  larger  Integration  and  Evaluation  Framework  NOPP  award.  A  site  for 
the  Coupling  Testbed  was  also  initiated  and  is  here: 
http://www.earthsystemcog.org/projects/couplingtestbed/ 

Initial  roadmaps  were  created  for  each  of  the  three  objectives,  and  are  linked  under  the  Roadmap  tabs 
on  the  two  sites  above.  The  roadmaps  include  status  at  start,  team  members,  desired  outcomes,  and  a 
sequence  of  tasks. 

1 .  Update  Numerical  Libraries  in  ESMF 

http://www.earthsystemcog.org/projects/espc-infrastmcture/moab_in_esmf 
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2.  Optimize  Component  Architectures  for  GPUs 

http;//www.earthsystemcog.org/projects/couplingtestbed/acceleratorplans 

3.  Couple  HYCOM  to  CESM 

(Part  1)  Introducing  NUOPC  conventions  into  CESM 

http  ;//www. earthsystemcog.org/projects/espc-infrastructure/nuopc_in_cesm 

(Part  2)  HYCOM  in  CESM  scientific  validation 

http://www.earthsystemcog.org/projects/espc-infrastructure/hycom_in_cesm_science 

The  roadmaps  are  being  updated  by  team  members  prior  to  the  weekly  telecons,  and  progress  and 
issues  are  discussed  during  the  calls.  Where  appropriate,  test  reports  describing  how  a  code  base  was 
validated  are  linked  to  the  roadmaps.  These  test  reports  include  information  that  supports  the 
reproducibility  of  results. 

Work  to  date  has  focused  mainly  on  the  third  item,  coupling  HYCOM  to  CESM. 

WORK  COMPLETED 

The  Roadmap  pages  listed  under  Approach  contain  detailed  descriptions  of  the  work  completed.  See 
summaries  below. 

1 .  Update  Numerical  Libraries  in  ESMF 

Walter  Spector  integrated  the  MOAB  software  into  the  ESMF  build  and  tested  it  on  about  14 
different  platform/compiler  combinations.  He  wrote  a  unit  test  so  that  checking  the  correct 
operation  of  MOAB  is  part  of  standard  ESMF  regression  testing. 

2.  Optimize  Component  Architectures  for  GPUs 

This  part  of  the  project  has  not  been  started  yet. 

3.  Couple  HYCOM  to  CESM 

Work  began  with  introducing  the  NUOPC  conventions  into  the  “dead”  model  configuration  of 
CESM.  The  dead  configuration  does  not  generate  valid  data,  but  exercises  the  coupler  internals 
and  is  used  for  testing.  The  dead  NUOPC  CESM  configuration  will  serve  as  a  starting  point  for 
an  active  ocean-only  configuration.  HYCOM  will  be  the  ocean  and  all  other  components  will 
be  run  in  a  “data”  mode  where  they  provide  only  data  inputs.  This  ocean-only  CESM 
configuration  will  be  tested  for  correctness  against  a  HYCOM  code  running  outside  of  the 
CESM.  Additional  active  components  will  be  introduced  incrementally,  beginning  with  the  ice 
model. 

There  are  two  distinct  parts  to  this  effort:  introducing  NUOPC  conventions  into  CESM  and 
setting  up  baselines  (and  later,  experiments)  for  scientific  validation  of  HYCOM. 

(Part  1)  Introducing  NUOPC  conventions  into  CESM 

Kathy  Saint  modified  the  CESM  driver  so  that  it  implements  the  NUOPC  driver,  wrapped  all 
the  CESM  dead  components  with  NUOPC  wrappers,  confirmed  correct  operation,  and 
generated  a  test  report.  Next,  Kathy  put  NUOPC  wrappers  on  each  of  the  data  and  active 
components.  She  then  added  the  CESM  exchange  fields  to  the  NUOPC  field  dictionary,  which 
is  used  to  check  that  fields  passed  between  components  match  properly.  Next  steps  will  likely 
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include  filling  out  the  NUOPC  CESM  configuration  with  NUOPC  generic  connector  and 
mediator  components,  and  validating  the  NUOPC  version  of  CESM  in  the  data  mode. 

A  synergistic  activity  (not  funded  under  01 for  ESPC)  is  being  undertaken  by  the  NCAR 
CESM  team.  It  involves  restructuring  the  CESM  driver  based  on  an  abstraction  representing 
the  component  type.  This  abstraction  subsumes  both  the  physical  domain  of  the  component 
and  the  framework  that  the  component  is  written  in.  It  allows  the  calling  sequence  in  the  driver 
to  be  simple  and  clear,  and  enables  options  for  MCT-based  component  types  and  ESME-based 
component  types  to  be  handled  in  a  concise  fashion.  In  addition,  the  CESM  team  developed  a 
strategy  for  referencing  memory  between  its  native  coupling  datatype  (MCT  Attribute  Vectors) 
and  ESME  Arrays.  This  will  reduce  the  copies  required  and  will  enable  ESME  and  MCT  to  be 
used  together  in  the  coupler. 

(Part  2)  PfYCOM  in  CESM  scientific  validation 

Alexandra  Bozec  has  been  working  on  setting  up  baseline  runs  of  HYCOM  with  data  ice.  The 
current  challenge  is  in  setting  up  the  ice  inputs  correctly,  so  that  expected  forms  of  quantities 
match. 

RESULTS 

The  most  tangible  results  to  date  are  two  test  reports  that  show  successful  completion  of  interim 
milestones. 

The  first  is  a  test  report  for  the  MOAB  build  within  ESME,  summarized  here: 
http://www.earthsystemcog.org/projects/nuopc/moab_build_in_esmf 
This  step  is  a  prerequisite  for  further  integration  of  MOAB  within  ESME. 

The  second  is  a  test  report  for  the  CESM  dead  configuration  with  a  NUOPC  driver,  summarized  here: 
http://www.earthsystemcog.org/projects/espc-infrastructure/nuopc_in_cesm_deadcomp_test_rpt 
This  step  represents  proof  of  concept  that  CESM  can  operate  as  a  NUOPC-based  system,  and  is  a  first 
step  toward  integration  of  HYCOM  into  CESM  using  NUOPC  infrastructure. 

There  are  important  advances  not  reflected  in  these  test  reports.  In  the  few  months  since  the  project 
started  (funds  became  available  in  July  2013)  there  has  been  excellent  progress  on  improving  the 
compatibility  of  NUOPC  and  CESM  code.  The  NCAR  team’s  redesign  of  the  CESM  coupler  and 
driver  has  meant  that  NUOPC  can  be  introduced  in  a  natural  and  clean  manner  within  CESM,  while 
preserving  the  capability  to  support  established  NCAR  and  DOE  infrastructure.  Eurther,  the  newly 
introduced  ability  within  CESM  to  share  memory  between  ESME  and  MCT  datatypes  makes  using 
both  tools  together  much  easier.  Using  both  is  appealing  because  there  are  functions  in  each  package 
that  the  other  does  not  contain.  This  ability  to  reference  memory  will  support  future  work  in  which 
MCT  and  ESME  anticipate  sharing  a  MOAB  mesh-based  data  representation. 

IMPACT/APPLICATIONS 

The  results  obtained  here  can  be  viewed  as  a  head  start  for  the  linked  proposals  funded  under  the  2013 
NOPP  AO  El  awards.  There  is  a  direct  correspondence  between  this  preliminary  work  and  the  AO  El 
project  An  Integration  and  Evaluation  Framework  for  ESPC  Coupled  Models  (PI:  Ben  Kirtman).  The 
Kirtman-led  project  also  focuses  on  a  coupled  CESM-HYCOM  configuration  and  builds  on  joint 
technical  work  involving  NCAR,  NRL,  DOE  laboratories,  and  ESME/NUOPC  developers. 
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The  work  presented  here  is  likely  to  inform  the  technical  strategies  adopted  by  other  NOPP  projects  as 
well.  For  example,  streamlining  the  CESM  coupler  while  introducing  NUOPC  code  is  enabling  ESMF 
data  structures  and  methods  that  support  in-memory,  during-run  grid  remapping  to  be  introduced  into 
CESM  in  an  efficient  way,  preserving  existing  infrastructure  where  it  makes  sense.  This  capability  is 
likely  to  be  a  first  step  in  coupling  components  with  adaptive  grids,  an  objective  of  the  NPS-NRE- 
Rice-UIUC  NOPP  AOEI  project. 

RELATED  PROJECTS 

The  OlforESPC  project  is  most  closely  related  to  the  projects  supported  under  the  2013  NOPP  AOEI 
awards,  listed  on  this  page;  http://coaps.fsu.edu/aoli/projects 

The  current  work  is  particularly  relevant  to  the  AOEI  project  An  Integration  and  Evaluation 
Framework  for  ESPC  Coupled  Models  (PI;  Ben  Kirtman),  which  also  focuses  on  a  CESM-HYCOM 
configuration  and  includes  joint  technical  work  involving  NCAR,  DOE  laboratories,  and 
ESME/NUOPC  developers. 
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